Methods and Systems for Gradually Adjusting Vehicle Sensor Perspective using Remote Assistance

ABSTRACT

Example embodiments relate to gradually adjusting a vehicle sensor perspective using remote assistance. A computing device may receive a request for assistance from a vehicle operating in an environment. The request indicates that the vehicle is stopped at a location with a sensor perspective of the environment that is at least partially occluded. Responsive to receiving the request for assistance, the computing device may display a graphical user interface (GUI) that represents a current state of the vehicle and includes a selectable option configured to enable the vehicle to gradually move forward a predefined distance. The computing device may detect a selection of the selectable option and transmit instructions that enable the vehicle to gradually move forward the predefined distance. The vehicle can then gradually move forward the predefined distance while also monitoring for one or more changes in the environment responsive to receiving the instructions.

BACKGROUND

Vehicles are used to complete various types of tasks, including objectand people transportation. With advances in technology, some vehiclesare configured with systems that enable the vehicles to operate in apartial or fully autonomous mode. When operating in a partial or fullyautonomous mode, some or all of the navigation aspects of vehicleoperation are controlled by a vehicle control system rather than atraditional human driver. Autonomous operation of a vehicle can involvesystems sensing the vehicle's surrounding environment to enable acomputing system to plan and safely navigate.

SUMMARY

Example embodiments describe techniques for gradually adjusting vehiclesensor perspective using remote assistance. The techniques may enableremote operators to provide assistance, which can enable autonomousvehicles to slowly adjust positions and overcome situations when thevehicle is stranded with vehicle sensors partially occluded andtherefore unable to gather enough information to determine a safenavigation strategy.

In one aspect, an example method is provided. The method may involvereceiving, at a computing device, a request for assistance from avehicle operating in an environment. The request indicates that thevehicle is stopped at a location with a sensor perspective of theenvironment that is at least partially occluded. The method may furtherinvolve, responsive to receiving the request for assistance, displaying,by the computing device, a graphical user interface (GUI) thatrepresents a current state of the vehicle and includes a selectableoption configured to enable the vehicle to gradually move forward apredefined distance. The method may also involve, based on detecting aselection of the selectable option, transmitting, by the computingdevice and to the vehicle, instructions that enable the vehicle togradually move forward the predefined distance. The vehicle isconfigured to gradually move forward the predefined distance while alsomonitoring for one or more changes in the environment responsive toreceiving the instructions.

In another aspect, an example system is provided. The system may includea vehicle and a computing device. The computing device is configured toreceive a request for assistance from the vehicle operating in anenvironment. The request indicates that the vehicle is stopped at alocation with a sensor perspective of the environment that is at leastpartially occluded. The computing device is further configured to,responsive to receiving the request for assistance, display a graphicaluser interface (GUI) that represents a current state of the vehicle andincludes a selectable option configured to enable the vehicle togradually move forward a predefined distance. The computing device isalso configured to, based on detecting a selection of the selectableoption, transmit, to the vehicle, instructions that enable the vehicleto gradually move forward the predefined distance. The vehicle isconfigured to gradually move forward the predefined distance while alsomonitoring for one or more changes in the environment responsive toreceiving the instructions.

In yet another example, an example non-transitory computer readablemedium having stored therein program instructions executable by acomputing system to cause the computing system to perform functions isprovided. The functions may include receiving a request for assistancefrom a vehicle operating in an environment. The request indicates thatthe vehicle is stopped at a location with a sensor perspective of theenvironment that is at least partially occluded. The functions may alsoinclude, responsive to receiving the request for assistance, displayinga graphical user interface (GUI) that represents a current state of thevehicle and includes a selectable option configured to enable thevehicle to gradually move forward a predefined distance. The functionsmay also include, based on detecting a selection of the selectableoption, transmitting, to the vehicle, instructions that enable thevehicle to gradually move forward the predefined distance. The vehicleis configured to gradually move forward the predefined distance whilealso monitoring for one or more changes in the environment responsive toreceiving the instructions.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the figures and the followingdetailed description.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a functional block diagram illustrating a vehicle, accordingto example implementations.

FIG. 2A illustrates a side view of a vehicle, according to one or moreexample embodiments.

FIG. 2B illustrates a top view of a vehicle, according to one or moreexample embodiments.

FIG. 2C illustrates a front view of a vehicle, according to one or moreexample embodiments.

FIG. 2D illustrates a back view of a vehicle, according to one or moreexample embodiments.

FIG. 2E illustrates an additional view of a vehicle, according to one ormore example embodiments.

FIG. 3 is a simplified block diagram for a computing system, accordingto one or more example embodiments.

FIG. 4 is a system for wireless communication between computing devicesand a vehicle, according to one or more example embodiments.

FIG. 5 illustrates a computing device displaying a graphical userinterface for enabling remote assistance, according to one or moreexample embodiments.

FIG. 6A illustrates a scenario involving a stranded vehicle with asensor perspective of the environment that is partially occluded,according to one or more example embodiments.

FIG. 6B illustrates a graphical user interface for providing remoteassistance to the vehicle shown in FIG. 6A, according to one or moreexample embodiments.

FIG. 6C illustrates another graphical user interface for providingremote assistance to the vehicle shown in FIG. 6A, according to one ormore example embodiments.

FIG. 6D further illustrates the scenario shown in FIG. 6A after thevehicle received remote assistance, according to one or more exampleembodiments.

FIG. 7A illustrates another scenario involving a vehicle requestingremote assistance, according to one or more example embodiments.

FIG. 7B illustrates the vehicle receiving remote assistance in thescenario shown in FIG. 7A, according to one or more example embodiments.

FIG. 7C further illustrates the scenario shown in in FIGS. 7A and 7Bwith the vehicle gradually adjusting sensor perspective based on remoteassistance, according to one or more example embodiments.

FIG. 8 is a flow chart of a method, according to exampleimplementations.

FIG. 9 is a schematic diagram of a computer program, according toexample implementations.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying figures, which form a part hereof. In the figures, similarsymbols typically identify similar components, unless context dictatesotherwise. The illustrative embodiments described in the detaileddescription, figures, and claims are not meant to be limiting. Otherembodiments may be utilized, and other changes may be made, withoutdeparting from the scope of the subject matter presented herein. It willbe readily understood that the aspects of the present disclosure, asgenerally described herein, and illustrated in the figures, can bearranged, substituted, combined, separated, and designed in a widevariety of different configurations, all of which are explicitlycontemplated herein.

Advancements in computing, sensors, and other technologies have enabledvehicles to safely navigate autonomously between locations withoutrequiring input from a driver. By processing measurements of thesurrounding environment from vehicle sensors in near real-time, anautonomous vehicle can transport passengers or objects between locationswhile avoiding obstacles, obeying traffic requirements, and performingother necessary actions that are typically conducted by a driver. Theshift of control of the vehicle over to a vehicle control system canpermit passengers to devote their attention to tasks other than driving.

The rise of autonomous-capable vehicles may contribute to an increaseusage of vehicle-sharing. Vehicle-sharing can often involve a fleet ofvehicles that are centrally-managed by an overall system that isconfigured to receive requests from potential passengers and dispatchnearby vehicles to locate, pick up, and provide rides to passengers inexchange for a fee. Within the fleet, each vehicle may fulfill requestsfor rides from passengers. For instance, a passenger may use asmartphone application or another computing device to submit a requestto the system managing the fleet of vehicles. A request from a potentialpassenger may include information to assist the system complete therequest, such as a current location of the passenger, a quantity ofpassengers that plan on receiving transportation, and one or moredesired destinations for the vehicle to drop off passengers. In responseto receiving the request, the system may identify and dispatch a vehiclewithin the fleet that can quickly travel to the passenger's currentlocation and autonomously transport the passenger or passengers to oneor more desired destinations accordingly. After completing a riderequest, the vehicle may subsequently be dispatched to providetransportation for other passengers.

Whether operating as part of a vehicle-sharing fleet or independently, avehicle capable of autonomous or semi-autonomous operation may encountersome situations where the vehicle systems can benefit from additionalassistance. In particular, the vehicle systems enabling autonomousoperation may request for remote assistance to overcome some potentialobstacles. A request for assistance may involve a vehicle communicationinterface (or another vehicle-based system) transmitting a request forassistance via wireless communication to a computing device positionedremotely from the vehicle. The request for assistance may include sensordata and other information that can help the remote computing device ora human operator use the computing device to provide some form ofassistance to the vehicle that may help the vehicle overcome the currentsituation. Further details regarding the communication of requests forassistance, information within a given request, and management ofrequests are disclosed herein.

Vehicles may submit requests for assistance to obtain human input tohelp resolve a variety of situations that a human driver may be able toovercome. In some instances, a vehicle autonomously navigating theenvironment may encounter situations where the vehicle's sensorperspective of the environment is partially occluded. Such a situationmay cause the vehicle to remain stopped until the environment changesenough to obtain sensor measurements for safely proceeding. For example,a vehicle stopped at an intersection may have a sensor perspective thatis partially occluded by parked cars, trees, signs, or other objects inthe environment. When a human driver encounters this and similarsituations, the driver might cause the vehicle to gradually move forwardsafely until the driver is safely able to see around the objects thatwere previously blocking the driver's perspective. While graduallymoving forward, the driver might stop the vehicle in response to theenvironment, such as to avoid another vehicle traveling proximate thevehicle. Because these actions by the driver can quickly overcome suchsituations, it is desirable to enable an autonomous vehicle to replicateactions similar to those performed by the driver in a safe manner.

Example embodiments described herein relate to systems and techniquesfor gradually adjusting vehicle sensor perspective using remoteassistance. When an autonomous vehicle encounters a situation wherenavigation progress is impeded by an occluded sensor perspective of theenvironment, the autonomous vehicle may obtain remote assistance thatcan help the vehicle quickly overcome the situation. Remote assistancetechniques described herein can be utilized in various situations, suchas when an autonomous vehicle is temporarily stranded at anintersection, a parking lot, in an area with road constructions, orother potential navigation environments.

To further illustrate, a remotely positioned computing device associatedwith a remote operator may initially receive a request for assistancefrom a vehicle operating in an environment. For instance, the vehiclemay be autonomously navigating a neighborhood or city and encounter asituation that vehicle systems could better manage with some remoteassistance. As such, the request for assistance received by thecomputing device may indicate details related to the vehicle'ssituation. For instance, the request may specify that the vehicle iscurrently stopped with a sensor perspective of the environment that isat least partially occluded by one or more objects, such as a parkedcar, a sign, shrubs, trees, pedestrians, or other objects. Responsive toreceiving the request from the vehicle, the computing device may providean interface (e.g., a graphical user interface (GUI)) for the humanoperator to review and subsequently provide assistance to the vehicle.Based on an input from the operator, the computing device may transmitinstructions to the vehicle.

The GUI generated by the computing device to enable remote assistancecan vary within embodiments. The GUI can be used to convey the situationencountered by the vehicle to the remote operator so that the remoteoperator can provide assistance to help resolve the situation. Forexample, the GUI may include images or other sensor data to helprepresent the situation encountered by the vehicle. The GUI can alsorepresent other information, such as information relating to the vehicle(e.g., location, quantity of passengers, type of vehicle).

In some examples, the GUI produced by the computing device enables inputfrom the remote operator. For instance, the computing device may providean interface configured with a single selectable option, which whenselected by the remote operator, causes the computing device to transmitinstructions that enable the vehicle to gradually move forward apredefined distance at or below a threshold speed (e.g., less than 5miles per hour (MPH)). Without such instructions from the remoteoperator, the vehicle's ability to modify its position may be limited.Upon reception of the instructions, however, the vehicle can beconfigured to gradually move forward the predefined distance based onthe instructions while also monitoring the environment for changes thatmay require the vehicle to stop gradually moving forward.

In some embodiments, an autonomous vehicle may be limited to a set ofoperations during navigation. As such, instructions that originate basedon an input or inputs from a remote operator may enable the vehicle toperform one or operations that are not included in the original set ofoperations. In other words, the operation set for an autonomous vehiclecan differ depending on if the vehicle is executing operations based oninstructions from a remote operator. For example, an autonomous vehiclemay lack the ability to independently gradually move forward orbackwards slowly at intersections to modify sensor perspective of theenvironment, but may be able to execute such operations upon receivinginstructions from a remote operator. Operators can also help identifyobjects in the environment, adjust navigation routes, confirm or denynavigation options proposed by a vehicle, check on passengers, andperform other forms of remote assistance.

Remote assistance for vehicles can originate from a network of remoteoperators. For example, a vehicle may submit a request for assistancethat is received at an entry point of the network. The entry point mayconnect the request with a remote operator that can provide assistance.The remote operator may be selected based on credentials associated withthe remote operator that indicate the operator expertise to handle thetype of assistance that is being requested and/or the operator'savailability, among other potential parameters. The entry point mayanalyze information within the request to route requests for assistanceaccordingly. For example, the network of remote operators may be used toprovide assistance to an entire fleet of autonomous vehicles.

Example systems within the scope of the present disclosure will now bedescribed in greater detail. An example system may be implemented in ormay take the form of an automobile, but other example systems can beimplemented in or take the form of other vehicles, such as cars, trucks,motorcycles, buses, boats, airplanes, helicopters, lawn mowers, earthmovers, boats, snowmobiles, aircraft, recreational vehicles, amusementpark vehicles, farm equipment, construction equipment, trams, golfcarts, trains, trolleys, and robot devices. Other vehicles are possibleas well.

Referring now to the figures, FIG. 1 is a functional block diagramillustrating vehicle 100, which represents a vehicle capable ofoperating fully or partially in an autonomous mode. More specifically,vehicle 100 may operate in an autonomous mode without human interaction(or reduced human interaction) through receiving control instructionsfrom a computing system (e.g., a vehicle control system). As part ofoperating in the autonomous mode, vehicle 100 may use sensors (e.g.,sensor system 104) to detect and possibly identify objects of thesurrounding environment to enable safe navigation. In someimplementations, vehicle 100 may also include subsystems that enable adriver (or a remote operator) to control operations of vehicle 100.

As shown in FIG. 1, vehicle 100 includes various subsystems, such aspropulsion system 102, sensor system 104, control system 106, one ormore peripherals 108, power supply 110, computer system 112, datastorage 114, and user interface 116. The subsystems and components ofvehicle 100 may be interconnected in various ways (e.g., wired or securewireless connections). In other examples, vehicle 100 may include moreor fewer subsystems. In addition, the functions of vehicle 100 describedherein can be divided into additional functional or physical components,or combined into fewer functional or physical components withinimplementations.

Propulsion system 102 may include one or more components operable toprovide powered motion for vehicle 100 and can include an engine/motor118, an energy source 119, a transmission 120, and wheels/tires 121,among other possible components. For example, engine/motor 118 may beconfigured to convert energy source 119 into mechanical energy and cancorrespond to one or a combination of an internal combustion engine, oneor more electric motors, steam engine, or Stirling engine, among otherpossible options. For instance, in some implementations, propulsionsystem 102 may include multiple types of engines and/or motors, such asa gasoline engine and an electric motor.

Energy source 119 represents a source of energy that may, in full or inpart, power one or more systems of vehicle 100 (e.g., engine/motor 118).For instance, energy source 119 can correspond to gasoline, diesel,other petroleum-based fuels, propane, other compressed gas-based fuels,ethanol, solar panels, batteries, and/or other sources of electricalpower. In some implementations, energy source 119 may include acombination of fuel tanks, batteries, capacitors, and/or flywheel.

Transmission 120 may transmit mechanical power from the engine/motor 118to wheels/tires 121 and/or other possible systems of vehicle 100. Assuch, transmission 120 may include a gearbox, a clutch, a differential,and a drive shaft, among other possible components. A drive shaft mayinclude axles that connect to one or more wheels/tires 121.

Wheels/tires 121 of vehicle 100 may have various configurations withinexample implementations. For instance, vehicle 100 may exist in aunicycle, bicycle/motorcycle, tricycle, or car/truck four-wheel format,among other possible configurations. As such, wheels/tires 121 mayconnect to vehicle 100 in various ways and can exist in differentmaterials, such as metal and rubber.

Sensor system 104 can include various types of sensors, such as GlobalPositioning System (GPS) 122, inertial measurement unit (IMU) 124, oneor more radar units 126, laser rangefinder/LIDAR unit 128, camera 130,steering sensor 123, and throttle/brake sensor 125, among other possiblesensors. In some implementations, sensor system 104 may also includesensors configured to monitor internal systems of the vehicle 100 (e.g.,O₂ monitors, fuel gauge, engine oil temperature, condition of brakes).

GPS 122 may include a transceiver operable to provide informationregarding the position of vehicle 100 with respect to the Earth. IMU 124may have a configuration that uses one or more accelerometers and/orgyroscopes and may sense position and orientation changes of vehicle 100based on inertial acceleration. For example, IMU 124 may detect a pitchand yaw of the vehicle 100 while vehicle 100 is stationary or in motion.

Radar unit 126 may represent one or more systems configured to use radiosignals to sense objects (e.g., radar signals), including the speed andheading of the objects, within the local environment of vehicle 100. Assuch, radar unit 126 may include one or more radar units equipped withone or more antennas configured to transmit and receive radar signals asdiscussed above. In some implementations, radar unit 126 may correspondto a mountable radar system configured to obtain measurements of thesurrounding environment of vehicle 100. For example, radar unit 126 caninclude one or more radar units configured to couple to the underbody ofa vehicle.

Laser rangefinder/LIDAR 128 may include one or more laser sources, alaser scanner, and one or more detectors, among other system components,and may operate in a coherent mode (e.g., using heterodyne detection) orin an incoherent detection mode. Camera 130 may include one or moredevices (e.g., still camera or video camera) configured to captureimages of the environment of vehicle 100.

Steering sensor 123 may sense a steering angle of vehicle 100, which mayinvolve measuring an angle of the steering wheel or measuring anelectrical signal representative of the angle of the steering wheel. Insome implementations, steering sensor 123 may measure an angle of thewheels of the vehicle 100, such as detecting an angle of the wheels withrespect to a forward axis of the vehicle 100. Steering sensor 123 mayalso be configured to measure a combination (or a subset) of the angleof the steering wheel, electrical signal representing the angle of thesteering wheel, and the angle of the wheels of vehicle 100.

Throttle/brake sensor 125 may detect the position of either the throttleposition or brake position of vehicle 100. For instance, throttle/brakesensor 125 may measure the angle of both the gas pedal (throttle) andbrake pedal or may measure an electrical signal that could represent,for instance, the angle of the gas pedal (throttle) and/or an angle of abrake pedal. Throttle/brake sensor 125 may also measure an angle of athrottle body of vehicle 100, which may include part of the physicalmechanism that provides modulation of energy source 119 to engine/motor118 (e.g., a butterfly valve or carburetor). Additionally,throttle/brake sensor 125 may measure a pressure of one or more brakepads on a rotor of vehicle 100 or a combination (or a subset) of theangle of the gas pedal (throttle) and brake pedal, electrical signalrepresenting the angle of the gas pedal (throttle) and brake pedal, theangle of the throttle body, and the pressure that at least one brake padis applying to a rotor of vehicle 100. In other embodiments,throttle/brake sensor 125 may be configured to measure a pressureapplied to a pedal of the vehicle, such as a throttle or brake pedal.

Control system 106 may include components configured to assist innavigating vehicle 100, such as steering unit 132, throttle 134, brakeunit 136, sensor fusion algorithm 138, computer vision system 140,navigation/pathing system 142, and obstacle avoidance system 144. Morespecifically, steering unit 132 may be operable to adjust the heading ofvehicle 100, and throttle 134 may control the operating speed ofengine/motor 118 to control the acceleration of vehicle 100. Brake unit136 may decelerate vehicle 100, which may involve using friction todecelerate wheels/tires 121. In some implementations, brake unit 136 mayconvert kinetic energy of wheels/tires 121 to electric current forsubsequent use by a system or systems of vehicle 100.

Sensor fusion algorithm 138 may include a Kalman filter, Bayesiannetwork, or other algorithms that can process data from sensor system104. In some implementations, sensor fusion algorithm 138 may provideassessments based on incoming sensor data, such as evaluations ofindividual objects and/or features, evaluations of a particularsituation, and/or evaluations of potential impacts within a givensituation.

Computer vision system 140 may include hardware and software operable toprocess and analyze images in an effort to determine objects,environmental objects (e.g., stop lights, road way boundaries, etc.),and obstacles. As such, computer vision system 140 may use objectrecognition, Structure From Motion (SFM), video tracking, and otheralgorithms used in computer vision, for instance, to recognize objects,map an environment, track objects, estimate the speed of objects, etc.

Navigation/pathing system 142 may determine a driving path for vehicle100, which may involve dynamically adjusting navigation duringoperation. As such, navigation/pathing system 142 may use data fromsensor fusion algorithm 138, GPS 122, and maps, among other sources tonavigate vehicle 100. Obstacle avoidance system 144 may evaluatepotential obstacles based on sensor data and cause systems of vehicle100 to avoid or otherwise negotiate the potential obstacles.

As shown in FIG. 1, vehicle 100 may also include peripherals 108, suchas wireless communication system 146, touchscreen 148, microphone 150,and/or speaker 152. Peripherals 108 may provide controls or otherelements for a user to interact with user interface 116. For example,touchscreen 148 may provide information to users of vehicle 100. Userinterface 116 may also accept input from the user via touchscreen 148.Peripherals 108 may also enable vehicle 100 to communicate with devices,such as other vehicle devices.

Wireless communication system 146 may securely and wirelesslycommunicate with one or more devices directly or via a communicationnetwork. For example, wireless communication system 146 could use 3Gcellular communication, such as CDMA, EVDO, GSM/GPRS, or 4G cellularcommunication, such as WiMAX or LTE. Alternatively, wirelesscommunication system 146 may communicate with a wireless local areanetwork (WLAN) using WiFi or other possible connections. Wirelesscommunication system 146 may also communicate directly with a deviceusing an infrared link, Bluetooth, or ZigBee, for example. Otherwireless protocols, such as various vehicular communication systems, arepossible within the context of the disclosure. For example, wirelesscommunication system 146 may include one or more dedicated short-rangecommunications (DSRC) devices that could include public and/or privatedata communications between vehicles and/or roadside stations.

Vehicle 100 may include power supply 110 for powering components. Powersupply 110 may include a rechargeable lithium-ion or lead-acid batteryin some implementations. For instance, power supply 110 may include oneor more batteries configured to provide electrical power. Vehicle 100may also use other types of power supplies. In an exampleimplementation, power supply 110 and energy source 119 may be integratedinto a single energy source.

Vehicle 100 may also include computer system 112 to perform operations,such as operations described therein. As such, computer system 112 mayinclude at least one processor 113 (which could include at least onemicroprocessor) operable to execute instructions 115 stored in anon-transitory computer readable medium, such as data storage 114. Insome implementations, computer system 112 may represent a plurality ofcomputing devices that may serve to control individual components orsubsystems of vehicle 100 in a distributed fashion.

In some implementations, data storage 114 may contain instructions 115(e.g., program logic) executable by processor 113 to execute variousfunctions of vehicle 100, including those described above in connectionwith FIG. 1. Data storage 114 may contain additional instructions aswell, including instructions to transmit data to, receive data from,interact with, and/or control one or more of propulsion system 102,sensor system 104, control system 106, and peripherals 108.

In addition to instructions 115, data storage 114 may store data such asroadway maps, path information, among other information. Suchinformation may be used by vehicle 100 and computer system 112 duringthe operation of vehicle 100 in the autonomous, semi-autonomous, and/ormanual modes.

Vehicle 100 may include user interface 116 for providing information toor receiving input from a user of vehicle 100. User interface 116 maycontrol or enable control of content and/or the layout of interactiveimages that could be displayed on touchscreen 148. Further, userinterface 116 could include one or more input/output devices within theset of peripherals 108, such as wireless communication system 146,touchscreen 148, microphone 150, and speaker 152.

Computer system 112 may control the function of vehicle 100 based oninputs received from various subsystems (e.g., propulsion system 102,sensor system 104, and control system 106), as well as from userinterface 116. For example, computer system 112 may utilize input fromsensor system 104 in order to estimate the output produced by propulsionsystem 102 and control system 106. Depending upon the embodiment,computer system 112 could be operable to monitor many aspects of vehicle100 and its subsystems. In some embodiments, computer system 112 maydisable some or all functions of the vehicle 100 based on signalsreceived from sensor system 104.

The components of vehicle 100 could be configured to work in aninterconnected fashion with other components within or outside theirrespective systems. For instance, in an example embodiment, camera 130could capture a plurality of images that could represent informationabout a state of an environment of vehicle 100 operating in anautonomous mode. The state of the environment could include parametersof the road on which the vehicle is operating. For example, computervision system 140 may be able to recognize the slope (grade) or otherfeatures based on the plurality of images of a roadway. Additionally,the combination of GPS 122 and the features recognized by computervision system 140 may be used with map data stored in data storage 114to determine specific road parameters. Further, radar unit 126 may alsoprovide information about the surroundings of the vehicle.

In other words, a combination of various sensors (which could be termedinput-indication and output-indication sensors) and computer system 112could interact to provide an indication of an input provided to controla vehicle or an indication of the surroundings of a vehicle.

In some embodiments, computer system 112 may make a determination aboutvarious objects based on data that is provided by systems other than theradio system. For example, vehicle 100 may have lasers or other opticalsensors configured to sense objects in a field of view of the vehicle.Computer system 112 may use the outputs from the various sensors todetermine information about objects in a field of view of the vehicle,and may determine distance and direction information to the variousobjects. Computer system 112 may also determine whether objects aredesirable or undesirable based on the outputs from the various sensors.In addition, vehicle 100 may also include telematics control unit (TCU)160. TCU 160 may enable vehicle connectivity and internal passengerdevice connectivity through one or more wireless technologies.

Although FIG. 1 shows various components of vehicle 100, i.e., wirelesscommunication system 146, computer system 112, data storage 114, anduser interface 116, as being integrated into the vehicle 100, one ormore of these components could be mounted or associated separately fromvehicle 100. For example, data storage 114 could, in part or in full,exist separate from vehicle 100. Thus, vehicle 100 could be provided inthe form of device elements that may be located separately or together.The device elements that make up vehicle 100 could be communicativelycoupled together in a wired and/or wireless fashion.

FIGS. 2A, 2B, 2C, 2D, and 2E illustrate different views of a physicalconfiguration of vehicle 100. The various views are included to depictexample sensor positions 202, 204, 206, 208, 210 on vehicle 100. Inother examples, sensors can have different positions on vehicle 100.Although vehicle 100 is depicted in FIGS. 2A-2E as a van, vehicle 100can have other configurations within examples, such as a truck, a car, asemi-trailer truck, a motorcycle, a bus, a shuttle, a golf cart, anoff-road vehicle, robotic device, or a farm vehicle, among otherpossible examples.

As discussed above, vehicle 100 may include sensors coupled at variousexterior locations, such as sensor positions 202-210. Vehicle sensorsinclude one or more types of sensors with each sensor configured tocapture information from the surrounding environment or perform otheroperations (e.g., communication links, obtain overall positioninginformation). For example, sensor positions 202-210 may serve aslocations for any combination of one or more cameras, radars, LIDARs,range finders, radio devices (e.g., Bluetooth and/or 802.11), andacoustic sensors, among other possible types of sensors.

When coupled at the example sensor positions 202-210 shown in FIGS.2A-2E, various mechanical fasteners may be used, including permanent ornon-permanent fasteners. For example, bolts, screws, clips, latches,rivets, anchors, and other types of fasteners may be used. In someexamples, sensors may be coupled to the vehicle using adhesives. Infurther examples, sensors may be designed and built as part of thevehicle components (e.g., parts of the vehicle mirrors).

In some implementations, one or more sensors may be positioned at sensorpositions 202-210 using movable mounts operable to adjust theorientation of one or more sensors. A movable mount may include arotating platform that can rotate sensors so as to obtain informationfrom multiple directions around vehicle 100. For instance, a sensorlocated at sensor position 202 may use a movable mount that enablesrotation and scanning within a particular range of angles and/orazimuths. As such, vehicle 100 may include mechanical structures thatenable one or more sensors to be mounted on top the roof of vehicle 100.Additionally, other mounting locations are possible within examples. Insome situations, sensors coupled at these locations can provide datathat can be used by a remote operator to provide assistance to vehicle100.

FIG. 3 is a simplified block diagram exemplifying computing device 300,illustrating some of the components that could be included in acomputing device arranged to operate in accordance with the embodimentsherein. Computing device 300 could be a client device (e.g., a deviceactively operated by a user (e.g., a remote operator)), a server device(e.g., a device that provides computational services to client devices),or some other type of computational platform. In some embodiments,computing device 300 may be implemented as computer system 112, whichcan be located on vehicle 100 and perform processing operations relatedto vehicle operations. For example, computing device 300 can be used toprocess sensor data received from sensor system 104. Alternatively,computing device 300 can be located remotely from vehicle 100 andcommunicate via secure wireless communication. For example, computingdevice 300 may operate as a remotely positioned device that a remotehuman operator can use to communicate with one or more vehicles.

In the example embodiment shown in FIG. 3, computing device 300 includesprocessing system 302, memory 304, input/output unit 306 and networkinterface 308, all of which may be coupled by a system bus 310 or asimilar mechanism. In some embodiments, computing device 300 may includeother components and/or peripheral devices (e.g., detachable storage,sensors, and so on).

Processing system 302 may be one or more of any type of computerprocessing element, such as a central processing unit (CPU), aco-processor (e.g., a mathematics, graphics, or encryptionco-processor), a digital signal processor (DSP), a network processor,and/or a form of integrated circuit or controller that performsprocessor operations. In some cases, processing system 302 may be one ormore single-core processors. In other cases, processing system 302 maybe one or more multi-core processors with multiple independentprocessing units. Processing system 302 may also include register memoryfor temporarily storing instructions being executed and related data, aswell as cache memory for temporarily storing recently-used instructionsand data.

Memory 304 may be any form of computer-usable memory, including but notlimited to random access memory (RAM), read-only memory (ROM), andnon-volatile memory. This may include flash memory, hard disk drives,solid state drives, re-writable compact discs (CDs), re-writable digitalvideo discs (DVDs), and/or tape storage, as just a few examples.

Computing device 300 may include fixed memory as well as one or moreremovable memory units, the latter including but not limited to varioustypes of secure digital (SD) cards. Thus, memory 304 can represent bothmain memory units, as well as long-term storage. Other types of memorymay include biological memory.

Memory 304 may store program instructions and/or data on which programinstructions may operate. By way of example, memory 304 may store theseprogram instructions on a non-transitory, computer-readable medium, suchthat the instructions are executable by processing system 302 to carryout any of the methods, processes, or operations disclosed in thisspecification or the accompanying drawings.

As shown in FIG. 3, memory 304 may include firmware 314A, kernel 314B,and/or applications 314C. Firmware 314A may be program code used to bootor otherwise initiate some or all of computing device 300. Kernel 314Bmay be an operating system, including modules for memory management,scheduling and management of processes, input/output, and communication.Kernel 314B may also include device drivers that allow the operatingsystem to communicate with the hardware modules (e.g., memory units,networking interfaces, ports, and busses), of computing device 300.Applications 314C may be one or more user-space software programs, suchas web browsers or email clients, as well as any software libraries usedby these programs. In some examples, applications 314C may include oneor more neural network applications and other deep learning-basedapplications. Memory 304 may also store data used by these and otherprograms and applications.

Input/output unit 306 may facilitate user and peripheral deviceinteraction with computing device 300 and/or other computing systems.Input/output unit 306 may include one or more types of input devices,such as a keyboard, a mouse, one or more touch screens, sensors,biometric sensors, and so on. Similarly, input/output unit 306 mayinclude one or more types of output devices, such as a screen, monitor,printer, speakers, and/or one or more light emitting diodes (LEDs).Additionally or alternatively, computing device 300 may communicate withother devices using a universal serial bus (USB) or high-definitionmultimedia interface (HDMI) port interface, for example. In someexamples, input/output unit 306 can be configured to receive data fromother devices. For instance, input/output unit 306 may receive sensordata from vehicle sensors.

As shown in FIG. 3, input/output unit 306 includes GUI 312, which can beconfigured to provide information to a remote operator or another user.GUI 312 may be displayable one or more display interfaces, or anothertype of mechanism for conveying information and receiving inputs. Insome examples, the representation of GUI 312 may differ depending on avehicle situation. For example, computing device 300 may provideinterface 312 in a particular format, such as a format with a singleselectable option for a remote operator to select from.

Network interface 308 may take the form of one or more wirelineinterfaces, such as Ethernet (e.g., Fast Ethernet, Gigabit Ethernet, andso on). Network interface 308 may also support communication over one ormore non-Ethernet media, such as coaxial cables or power lines, or overwide-area media, such as Synchronous Optical Networking (SONET) ordigital subscriber line (DSL) technologies. Network interface 308 mayadditionally take the form of one or more wireless interfaces, such asIEEE 802.11 (Wifi), BLUETOOTH®, global positioning system (GPS), or awide-area wireless interface. However, other forms of physical layerinterfaces and other types of standard or proprietary communicationprotocols may be used over network interface 308. Furthermore, networkinterface 308 may comprise multiple physical interfaces. For instance,some embodiments of computing device 300 may include Ethernet,BLUETOOTH®, and Wifi interfaces. In some embodiments, network interface308 may enable computing device 300 to connect with one or more vehiclesto allow for remote assistance techniques presented herein.

In some embodiments, one or more instances of computing device 300 maybe deployed to support a clustered architecture. The exact physicallocation, connectivity, and configuration of these computing devices maybe unknown and/or unimportant to client devices. Accordingly, thecomputing devices may be referred to as “cloud-based” devices that maybe housed at various remote data center locations. In addition,computing device 300 may enable the performance of embodiments describedherein, including efficient assignment and processing of sensor data.

FIG. 4 is a system for wireless communication between computing devicesand a vehicle, according to one or more example embodiments.Particularly, system 400 is shown with vehicle 402, remote computingdevice 404, and server 406 communicating wirelessly via network 408.System 400 may include other components not shown within otherembodiments, such as firewalls and multiple networks, among others.

Vehicle 402 may transport passengers or objects between locations, andmay take the form of any one or more of the vehicles discussed above,including passenger vehicles, cargo shipping vehicles, farming andmanufacturing vehicles, and dual-purpose vehicles. When operating in anautonomous mode, vehicle 402 may navigate with or without passengersenabling vehicle 402 to pick up and drop off passengers (or cargo)between desired destinations. In some embodiments, vehicle 402 canoperate as part of a fleet configured to communicate with a remoteoperator that may use remote computing device.

Remote computing device 404 may represent any type of device related toremote assistance techniques, including but not limited to thosedescribed herein. Within examples, remote computing device 404 mayrepresent any type of device configured to (i) receive informationrelated to vehicle 402, (ii) provide an interface (e.g., a GUI, physicalinput interfaces) through which a human operator can in turn perceivethe information and input a response related to the information, and(iii) transmit the response to vehicle 402 or to other devices (e.g.,storage at server 406). As such, remote computing device 404 may takevarious forms, such as a workstation, a desktop computer, a laptop, atablet, a mobile phone (e.g., a smart phone), a wearable device (e.g., aheadset) and/or a server. In some examples, remote computing device 404may include multiple computing devices operating together in a networkconfiguration. In further embodiments, remote computing device 404 mayresemble a vehicle simulation center with the remote operator positionedas the drive of the simulation center.

The position of remote computing device 404 relative to vehicle 402 canvary within examples. For instance, remote computing device 404 may havea remote position from vehicle 402, such as operating inside a physicalbuilding. In another example, remote computing device 404 may correspondto a computing device within vehicle 402 that is physically separatefrom vehicle 402, but with which a human operator can interact while asa passenger or driver of vehicle 402. In some examples, remote computingdevice 404 may be a computing device with a touchscreen operable by thepassenger of vehicle 402.

In some implementations, operations described herein that are performedby remote computing device 404 may be additionally or alternativelyperformed by vehicle 402 (i.e., by any system(s) or subsystem(s) ofvehicle 200). In other words, vehicle 402 may be configured to provide aremote assistance mechanism with which a driver or passenger of thevehicle can interact.

In addition, operations described herein can be performed by any of thecomponents communicating via network 408. Particularly, remote computingdevice 404 may determine remote assist options for a human operator toreview based on different levels of information provided by vehicle 402.In some embodiments, vehicle 402 may determine potential options forremote computing device 404 to display. Potential options could includeroutes, vehicle movements, and other navigation parameters for review byremote computing device 404 and/or a remote operator using remotecomputing device 404. For example, vehicle 402 may determine that it iscurrently stranded (e.g., stopped) with a sensor perspective ispartially blocked by objects preventing vehicle 402 from having a clearenough understanding of the environment to further navigate andsubsequently request for remote assistance from a remote operator.Vehicle 402 may provide an option to allow remote operator to enablevehicle to creep forward (e.g., move forward gradually at or below athreshold speed) for a predefined distance to modify its sensorperspective of the environment.

In other embodiments, remote computing device 404 may analyze sensordata or other information from vehicle 402 to determine the situationand potential options for a remote operator to review. For instance,remote computing device 404 may determine a route and/or operations forvehicle 402 to execute using information from vehicle 402 and/or otherexternal sources (e.g., server 406). In some embodiments, remotecomputing device 404 may generate a GUI to display one or moreselectable options for review by a remote operator.

Server 406 may be configured to wirelessly communicate with remotecomputing system 404 and vehicle 402 via network 408 (or perhapsdirectly with remote computing system 404 and/or vehicle 402). As such,server 406 may represent any computing device configured to receive,store, determine, and/or send information relating to vehicle 402 andthe remote assistance thereof. As such, server 406 may be configured toperform any operation(s), or portions of such operation(s), that is/aredescribed herein as performed by remote computing system 404 and/orvehicle 402. Some implementations of wireless communication related toremote assistance may utilize server 406, while others may not.

Network 408 represents infrastructure that can enable wirelesscommunication between computing devices, such as vehicle, 402, remotecomputing device 404, and server 406. For example, network 408 cancorrespond to a wireless communication network, such as the Internet ora cellular wireless communication network. The various systems describedabove may perform various operations. These operations and relatedfeatures will now be described.

In some examples, a remote computing system (e.g., remote computingsystem 404 or server 406) may operate in one of two modes. The first ofthese modes may serve, in essence, as a means for a human operator (ofthe vehicle and/or the remote computing system) to provide remoteassistance support for the vehicle. The remote computing system mayenable a human operator to provide this support in near real-time orless frequently than real-time.

The second of these two modes may serve as a means for keeping the humanoperator alert. The human operator may be a passenger or driver of thevehicle, or may be a third party located remotely from the vehicle buttasked with the responsibility of providing remote assistance to thevehicle (and possibly to other vehicles as well). Regardless of who thehuman operator is, it is desirable to keep the human operator alert sothat the human operator can provide optimal remote assistance withminimal delay.

For instance, there may be scenarios in which the vehicle may not haverequested remote assistance in a certain amount of time (e.g., onehour), and therefore the human operator tasked with providing remoteassistance to the vehicle may not have taken any remote assistanceaction in that amount of time, which may be long enough where the humanoperator may become fatigued or otherwise less attentive than desirable.In these and other types of possible scenarios, it may be desirable toperiodically prompt the human operator during this time, via the remotecomputing system, with alertness data to keep them alert. The alertnessdata may take various forms, such as archived images, audio, or videohaving confirmed or unconfirmed object identifications, also includinggenerated natural-language questions regarding the confirmed orunconfirmed object identifications.

Remote assistance tasks may also include the human operator providing aninstruction to control operation of the vehicle (e.g., instruct thevehicle to travel to a particular destination associated with anidentified passenger). In some scenarios, the vehicle itself may controlits own operation based on the human operator's feedback related to theidentification of the object. For instance, upon receiving aconfirmation that the occupancy of the vehicle meets a desiredoccupancy, the vehicle control system may cause the vehicle to safelytransport the passengers to a requested destination.

In some examples, a remote operator can enable a vehicle to temporarilyperform one or more operations to resolve a situation that the vehiclemay normally not be permitted to perform. For instance, remote computingdevice 404 may be used to enable vehicle 402 to back up, navigate with adecreased buffer zone, or travel in a zone that is usually off limits(e.g., over the median or use a driveway). In some examples, a humanoperator may enable or instruct vehicle 402 to move forward one or moreshort distances at low speeds (e.g., under 5 MPH) to modify the sensorperspective of the environment of vehicle 402 or to modify its locationto resolve another issue (e.g., blocking a sidewalk).

FIG. 5 illustrates a computing device displaying a GUI for enablingremote assistance, according to one or more example embodiments. GUI 502is shown displayed by computing device 500 and includes representationof the environment 504, selectable option 506, and contextualinformation 508. In other embodiments, GUI 502 may include more or lesselements in other potential arrangements.

GUI 502 represents a system of interactive visual components forcomputer software. As such, GUI 502 can be used to display objects thatconvey information to a remote operator and also represent actions thatcan be taken by the remote operator. Computing device 500 can generateGUI 502 based on templates enabling an available remote operator toquickly review and provide assistance to a vehicle. Computing device maydisplay GUI 502 on a display interface, such as a touch screen. In otherexamples, computing device may display GUI 502 or elements from GUI 502via a display interface associated with a head-mounted wearablecomputing device (e.g., augmented reality).

Representation of the environment 504 is an object displayable via GUI502 that can represent the current environment of a vehicle. Bydisplaying representation of the environment 504, a remote operator mayreview the sensor perspective of the environment as captured by vehiclesensors. For instance, representation of the environment 504 may displayimages of the environment as captured by vehicle cameras. In otherinstances, sensor data from different types of sensors can be used togenerate and provide representation of the environment 504 via GUI 502.

In some examples, computing device 500 may further obtain map data basedon a location of the vehicle. For instance, the vehicle may provide GPSmeasurements or another indication of the vehicle's location. By usingthe vehicle's location, computing device 500 can acquire map data andfurther enhance the information included within representation of theenvironment 504 and/or other objects displayed via GUI 502. For example,computing device 500 can determine and display representation ofenvironment 504 as an elevated view of the vehicle and nearbysurroundings estimated based on the map data and the sensor data fromthe vehicle. In some examples, GUI 502 may include both a sensorperspective of the vehicle's environment and the elevated view estimatedbased on one or both of the sensor data and map data.

Representation of the environment 504 may also include predefineddistance indication 514, which can be used to represent the distancethat the remote operator may enable the vehicle to move gradually inorder to adjust the sensor perspective of the environment for vehiclesensors. Computing device 500 may determine the predefined distance invarious ways within examples. In one example, predefined distanceindication 514 may represent a predefined distance that is preset forcertain environment conditions. For instance, the predefined distancecan depend on the vehicle having a location proximate to anintersection. In another example, predefined distance indication 514 mayrepresent a predefined distance determined by computing device 500specifically for the vehicle's situation that requires assistance.Particularly, computing device 500 may utilize sensor data and/or asuggested distance obtained from the vehicle to determine a predefineddistance and subsequently represent that predefined distance usingpredefined distance indication 514.

Predefined distance indication 514 can be conveyed by GUI 502 in avariety of ways within examples. For instance, predefined distanceindication 514 can involve a virtual placement of a mark (e.g., a line,a distance indication) positioned virtually relative to the vehiclewithin representation of the environment 504. Alternatively, predefineddistance indication 514 can be shown as a distance without associationto representation of the environment. For example, predefined distanceindication 514 may propose enabling the vehicle to move one or twometers forward gradually. As indicated herein, moving forward graduallycan involve the vehicle navigating forward in an idling mode or at orbelow a threshold speed (e.g., below 2 miles per hour (MPH)), which canenable sensors to slowly change orientation and position relative toobjects in the environment. Such a creep forward technique can be usedat intersections to see around parked vehicles, signs, pedestrians, busstops, trees, shrubs, and other potential objects that can block sensormeasurements of an area in the environment. In addition, the techniquecan also be used in parking lots and other situations where navigatingthe vehicle slowly would help assist the vehicle further navigate.

In some examples, predefined distance 514 may involve a vehicle backingup slowly. Such a technique may similar adjust the vehicle's sensorperspective of the environment.

Selectable option 506 represents an element of GUI 502 that a remoteoperator may select to provide instructions to the vehicle that enablethe vehicle to gradually move forward the predefined distance, which maybe presented by predefined distance indication 514. Particularly,selectable option 506 can be a virtual button, mark, or an object havinganother form that allows the remote operator to easily and clearlyselect to enable the vehicle to subsequently creep forward to modify itssensor perspective of the environment. In the embodiment shown in FIG.5, selectable option 506 is the only option available for the selectionby the remote operator. Particularly, the layout of GUI 502 shown withselectable option 506 as the only option simplifies the review processfor the remote operator and also limits the instructions that the remoteoperator may choose from. In other examples, additional options may beincluded. For instance, in another embodiment, GUI 502 may display afirst option for a first predefined distance and a second option forenabling the vehicle to travel a second predefined distance. Such alayout can enable the remote operator to have more influence insubsequent operations of the vehicle.

GUI 502 also includes contextual information 508, which may conveyadditional information to supplement a remote operator's understandingof the vehicle's situation. As shown in FIG. 5, contextual information508 includes vehicle information 510 and location information 512.Vehicle information 510 may indicate a variety of information about thevehicle, such as the type of vehicle, the vehicle sensors on thevehicle, the quantity of the passengers, and target destination, etc.Location information 512 may represent information based on the currentlocation of the vehicle, such as map data depicting the environment.Contextual information 508 may also specify information related to thesituation, such as how long has the vehicle been stranded and a reasonproposed by the vehicle for the stranding.

FIGS. 6A, 6B, 6C and 6D illustrate a scenario involving a vehiclegradually adjusting vehicle sensor perspective using remote assistance,according to an example embodiment. In FIG. 6A, scenario 600 is shownfrom an environment perspective from the view point from behind vehicle602. As shown in scenario 600, vehicle 602 is stopped in front of stopsign 604 at a four way intersection and is requesting remote assistancedue to a sensor perspective of the environment that is partiallyoccluded by vehicle 608. In particular, the sensor perspective forvehicle 602 lacks a clear understanding of area 606 positioned behindvehicle 608 due to the presence of vehicle 608, which may be parked ortemporarily stopped at the location shown in FIG. 6A. The position andarrangement of elements within scenario 600 are shown to represent onepotential situation that can occur during autonomous navigation byvehicle 602.

Without enough measurements of area 606, vehicle 602 may be configuredto remain stopped until determining a safe navigation strategy. In someinstances, vehicle 608 may move and vehicle 602 may resume navigationusing new measurements of area 606. In other instances, after becomingstranded at the current location shown in FIG. 6A, vehicle 602 maytransmit a request for assistance to a remote operator. Particularly, avehicle system (e.g., the navigation system) from vehicle 602 maytransmit the request to a remote assistance network, which maysubsequently connect the vehicle system with the computing device of aremote operator that can promptly provide assistance to help vehiclesystems overcome the issue.

In some embodiments, vehicle systems may be configured to request remoteassistance after being stranded in the same location for a thresholdduration of time (e.g., 60 seconds). The threshold duration of time canvary within examples and may depend on external factors, such as thepresence of vehicles behind (or nearby) vehicle 602. For example, when avehicle is detected behind vehicle 602, the threshold duration forrequesting remote assistance may be shorter to avoid delaying thevehicle or vehicles waiting for vehicle 602 to move.

The request may include information that depicts the situationencountered by vehicle 602. For example, the request may include anindication that vehicle 602 is stopped at a particular location andinclude a sensor perspective of the environment as measured from thecurrent location of vehicle 602. The sensor perspective can include adifferent amount of information and measurements from one or more typesof sensors. In some examples, the sensor perspective can be conveyed asa 3D map of the environment generated by the sensor processing system ofthe vehicle using one or more types of sensors. The sensor perspectivecan include images or video from cameras, LIDAR measurements, radarmeasurements, GPS measurements, and motion measurements from inertialmeasurement unit (IMU).

As such, the computing device receiving the request for assistance mayresponsively generate a GUI that can allow for a remote operator toreview the situation and provide assistance. For example, the computingdevice may generate a GUI similar to GUI 502 shown in FIG. 5. The GUIcan convey sensor data in different arrangements and other informationrelated to the situation (e.g., map data).

FIG. 6B depicts a GUI for enabling remote assistance for the situationillustrated in FIG. 6A. Particularly, a computing device may cause GUI610 to display on a display interface, such as a touchscreen or a highdefinition (HD) display. GUI 610 represents the sensor perspective (or aportion of the sensor perspective) of the environment from vehicle.

GUI 610 also shows selectable option 612 and visual line 614 virtuallyplaced in the representation of the vehicle's sensor perspective.Selectable option 612 represents a remote assistance option for a remoteoperator to review to assist vehicle 602. As shown, selectable option612 states “SELECT TO ENABLE VEHICLE TO MOVE SLOWLY UP TO LINE,” with anarrow pointing to virtual line 614 but can differ in other examples. Forinstance, selectable option 614 may state “creep forward” or otherphrases. Selection option 612 may be touched if the remote operator isusing a touch screen, selected using a mouse or a pointer, picked via abutton, or requested in other ways within examples. For instance, aremote operator may speak a word or phrase into a microphone to provideremote assistance. In addition, the computing device may use audio tocommunicate options to a remote operator. For instance, the computingdevice may audibly communicate the phrase shown as selectable option 612using an audio output. In other embodiments, GUI 610 may enable a remoteoperator to merely draw virtual line 614 to cause vehicle 602 to slowlymove forward that predefined distance. In further examples, a creepforward technique can be selected via a button.

In further examples, GUI may further include other options. Forinstance, GUI 610 may include a request more information option. Theremote operator can use the option to obtain additional sensor data orcommunicate with a passenger.

The computing device may also perform image processing, computer vision,or other techniques to further enhance the remote operator'sunderstanding of the situation. For instance, virtual box 616 is showndrawn around a portion of vehicle 608 that is blocking measurements ofthe environment. By using virtual boxes (e.g., virtual box 616) andother visual indications (e.g., virtual line 614), a remote operator maybe informed which object is occluding the sensor perspective of theenvironment for vehicle 608 and also understand a strategy for resolvingthe issue. The remote operator can review GUI 610 and select selectableoption 612 to cause vehicle 602 to move slowly a predefined distanceforward represented virtually in GUI 610 by virtual line 614. Vehicle602 may receive the remote assistance and gradually move forward asinstructed while also monitoring for changes in the environment. Forinstance, sensors may detect vehicle 608 moved and responsively causenavigation systems to resume safe normal navigation without graduallymoving forward as instructed by the remote operator. In anotherinstances, sensors may detect other environment changes, such as a cardriving nearby, a pedestrian crossing the intersection, or animal in theroadway. As a result, vehicle 602 may stop to avoid one or more detectedobstacles and overrule the instructions.

In some embodiments, the computing device may use sensor data stored inmemory to display prior events involving the vehicle. For instance, thecomputing device may display sensor data captured as a vehicleapproached an intersection prior to becoming stranded due to objectsblocking further measurements of an area of the environment. Inaddition, the computing device may supply audio from passengers or ofthe external environment from vehicle microphones.

FIG. 6C depicts another GUI for enabling remote assistance for thescenario shown in FIG. 6A. GUI 620 represents an environment of thevehicle 602 from an elevated viewpoint (e.g., a bird's eye view). Acomputing device may obtain sensor data from vehicle 602 that includesthe location of vehicle 602. Using the location, the computing devicemay utilize map data that represents the location and the sensor data togenerate GUI 620 using the elevated perspective.

As shown, the elevated viewpoint shows vehicle 622 and vehicle 628,which represent the locations of vehicle 602 and vehicle 608,respectively. In addition, similar to GUI 610 shown in FIG. 6B, GUI 620includes selectable option 626 to represent how the remote operator canassist vehicle 602 and virtual box 630 boxed around vehicle 628 toconvey to the remote operator that vehicle 608 is blocking part of thesensor measurements of vehicle 602 in scenario 600. In some examples,GUI 620 or other GUIs described herein may further include text (or showtext if objects are selected) that indicates vehicle 608 (represented byvehicle 628 for GUI 620) is the source of the occlusion and cause of therequest for assistance by vehicle 602.

GUI 620 also includes virtual line 624 to represent a predefineddistance (e.g., a few centimeters or meters) that selectable option 626may cause vehicle 602 to move forward in the environment to graduallyadjust sensor perspective. The computing device may determine thepredefined distance based on positioning of vehicle 622 and vehicle 628represented via GUI 620 or other factors and responsively displayvirtual line 624 to represent the distance. In other examples, thepredefined distance may be preconfigured or preset by an administrator.In other examples, the remote operator may first draw virtual line 624and subsequently select selectable option 626 to confirm that virtualline 624 was accurately placed. Such a configuration may enable theremote operator to have greater influence on the operations of vehicle602.

FIG. 6D shows an indication of predefined distance 640 that vehicle 602is configured to gradually move in response to receiving correspondinginstructions from a remote operator. Particularly, predefined distance640 may represent a distance that vehicle 602 is configured to graduallymove after receiving instructions provided by a remote operator. Whilegradually moving forward predefined distance 640 towards a center of theintersection, the sensor perspective also gradually changes. In someinstances, the sensor perspective for vehicle 602 may adjust enough tocapture measurements of area 606 without interference from vehicle 608.

In other instances, however, vehicle 602 may travel predefined distance640 and its sensor perspective may still be occluded by vehicle 608. Insuch a situation, vehicle 602 may transmit another request foradditional assistance from the remote operator, who may in turn use theinterface provided by the computing device to instruct vehicle 602 tomove another predefined distance further to see if that helps improvethe vehicle's sensor perspective of the environment. In addition,vehicle 602 may continue to measure the environment prior to, during,and after moving forward to ensure safety. In some instances, theenvironment might change in a way that causes vehicle 602 to stopmoving. In such a situation, vehicle 602 may overrule instructions fromthe remote operator or temporarily pause performance of theinstructions. Vehicle 602 may request for further assistance if strandedagain and/or if enough time passed since receiving the originalassistance from the operator (e.g., a threshold period of time haspassed).

FIGS. 7A, 7B, and 7C illustrate a scenario involving a vehicle graduallyadjusting vehicle sensor perspective using remote assistance, accordingto one or more example embodiments. In FIG. 7A, scenario 700 showsvehicle 702 stopped with sensor perspective 706 of the environmentblocked by shrub 704. Vehicle 702 may be stopped because shrub 704 ispreventing vehicle 702 from determining if potential objects are in thatarea of the environment, such as a vehicle traveling down that road. Assuch, vehicle 702 may request remote assistance as described herein.

A remote operator may provide remote assistance to vehicle 702. Forinstance, FIG. 7B depicts predefined distance 708 positioned in front ofvehicle 702. Vehicle 702 may receive instructions from a remote operatorto slowly navigate forward for predefined distance 708 while monitoringthe environment. Predefined distance 708 can be preset or can bedetermined by either vehicle 702, the remote computing device, or theremote operator based on sensor data or other information.

FIG. 7C further shows vehicle 702 at a subsequent location after movingforward predefined distance 708. At the subsequent location, vehicle's702 new sensor perspective 710 includes measurements that avoid shrub704. Vehicle 702 may subsequently be able to navigate safely using newsensor perspective 710, which involves measurements of the roadpositioned behind shrub 704.

FIG. 8 is a flow chart of a method for using remote assistance togradually adjust the sensor perspective of a vehicle, according toexample implementations. Method 800 represents an example method thatmay include one or more operations, functions, or actions, as depictedby one or more of blocks 802, 804, and 806, each of which may be carriedout by any of the systems, devices, and/or vehicles shown in FIGS. 1-7C,among other possible systems. For instance, system 400 depicted in FIG.4 may enable execution of method 800.

Those skilled in the art will understand that the flowchart describedherein illustrates functionality and operations of certainimplementations of the present disclosure. In this regard, each block ofthe flowchart may represent a module, a segment, or a portion of programcode, which includes one or more instructions executable by one or moreprocessors for implementing specific logical functions or steps in theprocess. The program code may be stored on any type of computer readablemedium, for example, such as a storage device including a disk or harddrive.

In addition, each block may represent circuitry that is wired to performthe specific logical functions in the process. Alternativeimplementations are included within the scope of the exampleimplementations of the present application in which functions may beexecuted out of order from that shown or discussed, includingsubstantially concurrent or in reverse order, depending on thefunctionality involved, as would be understood by those reasonablyskilled in the art.

At block 802, method 800 involves receiving a request for assistancefrom a vehicle operating in an environment. In some examples, therequest indicates that the vehicle is stopped at a location with asensor perspective of the environment that is at least partiallyoccluded. The interface by one or more objects in the environment maylimit the vehicle's understanding of the surrounding environment. As aresult, the vehicle may remain stationary and request remote assistance.

In some examples, the vehicle may transmit the request after a thresholdduration of time has passed since the vehicle initially became stranded.For example, the vehicle may request assistance after 60 seconds haspassed because the environment might change during the 60 seconds thatenables the vehicle to safely proceed with navigation without remoteassistance.

At block 804, method 800 involves displaying a graphical user interface(GUI) that represents a current state of the vehicle and includes anselectable option configured to enable the vehicle to gradually moveforward a predefined distance. The current state of the vehicle may berepresented based on sensor data and other information received from thevehicle. In addition, the selectable option may depend on the particularsituation encountered by the vehicle.

In some examples, the computing device may display the GUI such that theselectable is selectable only for a threshold quantity of selectionswithin a predefined duration of time. For example, the computing devicemay use the limitation option to prevent a human operator from selectingthe option three or more times within a two minute window. In addition,the computing device may also receive a suggested distance for graduallymoving forward from the vehicle. Based on the suggested distance, thecomputing device may determine the predefined distance.

At block 806, method 800 involves transmitting, to the vehicle,instructions that enable the vehicle to gradually move forward thepredefined distance. The computing device may transmit the instructionsbased on detecting a selection of the selectable option. In particular,the remote operator may review the situation as presented via the GUIand subsequently select the selectable option to provide remoteassistance to the vehicle.

After receiving the instructions, the vehicle may be configured togradually move forward the predefined distance based on the instructionswhile also monitoring for one or more changes in the environment.Particularly, monitoring the environment using one or more vehiclesensors can enable vehicle systems to stop moving forward (or in anotherdirection) when needed to maintain safety.

In some examples, the computing device may also receive sensor datarepresenting the sensor perspective of the environment and furtherdisplay the sensor perspective of the environment based on the sensordata when displaying the GUI that represents the current state of thevehicle and includes the selectable option. In addition, the computingdevice may also obtain map data based on the location of the vehicle.Using the map data, the computing device may display an elevated view ofthe vehicle at the location estimated based on the map data and thesensor data. In some examples, the computing device may determine thepredefined distance based on one or both of the sensor data and the mapdata.

The computing device may receive a second request from the vehicle insome instances. The second request can include an indication that thevehicle is stopped at a second location with a second sensor perspectiveof the environment that is at least partially occluded after graduallymoving forward the predefined distance from the location. Responsive toreceiving the second request for assistance, the computing device maydisplay a second GUI that represents a new state of the vehicle andincludes a second selectable option configured to enable the vehicle togradually move forward a second predefined distance. In some instances,the second predefined distance is less than the predefined distance.

In some examples, the computing device may receive sensor datarepresenting the sensor perspective of the environment from the vehicle.The computing device may identify one or more objects causing the sensorperspective of the environment that is at least partially occluded basedon the sensor data. As such, the computing device may display the GUIwith the sensor perspective of the environment with boxes outlining theone or more objects identified as causing the occlusion. In addition,the computing device may also display the GUI such that the GUI includesa visual indication of the predefined distance positioned virtuallyrelative to the sensor perspective of the environment.

FIG. 9 is a schematic diagram of a computer program, according to anexample implementation. In some implementations, the disclosed methodsmay be implemented as computer program instructions encoded on anon-transitory computer-readable storage media in a machine-readableformat, or on other non-transitory media or articles of manufacture.

In the embodiment shown in FIG. 9, computer program product 900 isprovided using signal bearing medium 902, which may include one or moreprogramming instructions 904 that, when executed by one or moreprocessors may provide functionality or portions of the functionalitydescribed above with respect to FIGS. 1-7C.

Signal bearing medium 902 may encompass a non-transitorycomputer-readable medium 906, such as, but not limited to, a hard diskdrive, a Compact Disc (CD), a Digital Video Disk (DVD), a digital tape,memory, components to store remotely (e.g., on the cloud) etc. In someimplementations, signal bearing medium 902 may encompass computerrecordable medium 908, such as, but not limited to, memory, read/write(R/W) CDs, R/W DVDs, etc.

In some implementations, signal bearing medium 902 may encompasscommunications medium 910, such as, but not limited to, a digital and/oran analog communication medium (e.g., a fiber optic cable, a waveguide,a wired communications link, a wireless communication link, etc.).Similarly, signal bearing medium 902 may correspond to a remote storage(e.g., a cloud). A computing system may share information with thecloud, including sending or receiving information. For example, thecomputing system may receive additional information from the cloud toaugment information obtained from sensors or another entity. Thus, forexample, signal bearing medium 902 may be conveyed by a wireless form ofcommunications medium 910.

One or more programming instructions 904 may be, for example, computerexecutable and/or logic implemented instructions. In some examples, acomputing device such as computer system 112 shown in FIG. 1 orcomputing device 300 shown in FIG. 3 may be configured to providevarious operations, functions, or actions in response to programminginstructions 904 conveyed to the computer system by one or more ofcomputer readable medium 906, computer recordable medium 908, and/orcommunications medium 910. The non-transitory computer readable mediumcould also be distributed among multiple data storage elements and/orcloud (e.g., remotely), which could be remotely located from each other.Computing device that executes some or all of the stored instructionscould be a vehicle. Alternatively, the computing device that executessome or all of the stored instructions could be another computingdevice, such as a server.

The above detailed description describes various features and functionsof the disclosed systems, devices, and methods with reference to theaccompanying figures. While various aspects and embodiments have beendisclosed herein, other aspects and embodiments will be apparent. Thevarious aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims.

What is claimed is:
 1. A method comprising: receiving, at a computingdevice, a request for assistance from a vehicle operating in anenvironment, wherein the request indicates that the vehicle is stoppedat a location with a sensor perspective of the environment that is atleast partially occluded; responsive to receiving the request forassistance, displaying, by the computing device, a graphical userinterface (GUI) that represents a current state of the vehicle andincludes a selectable option configured to enable the vehicle togradually move forward a predefined distance; and based on detecting aselection of the selectable option, transmitting, by the computingdevice and to the vehicle, instructions that enable the vehicle togradually move forward the predefined distance, wherein the vehicle isconfigured to gradually move forward the predefined distance while alsomonitoring for one or more changes in the environment responsive toreceiving the instructions.
 2. The method of claim 1, wherein receivingthe request for assistance from the vehicle operating in the environmentfurther comprises: receiving sensor data representing the sensorperspective of the environment; and wherein displaying the GUI thatrepresents the current state of the vehicle and includes the selectableoption comprises: displaying the sensor perspective of the environmentbased on the sensor data.
 3. The method of claim 2, further comprising:obtaining map data based on the location of the vehicle; and displayingan elevated view of the vehicle at the location estimated based on themap data and the sensor data.
 4. The method of claim 3, furthercomprising: determining the predefined distance based on one or both ofthe sensor data and the map data.
 5. The method of claim 1, whereinreceiving the request for assistance from the vehicle operating in theenvironment comprises: receiving a suggested distance for graduallymoving forward from the vehicle; and based on the suggested distance,determining the predefined distance.
 6. The method of claim 1, furthercomprising: receiving, at the computing device, a second request forassistance from the vehicle, wherein the second request includes anindication that the vehicle is stopped at a second location with asecond sensor perspective of the environment that is at least partiallyoccluded after gradually moving forward the predefined distance from thelocation; and responsive to receiving the second request for assistance,displaying a second GUI that represents a new state of the vehicle andincludes a second selectable option configured to enable the vehicle togradually move forward a second predefined distance.
 7. The method ofclaim 6, wherein the second predefined distance is less than thepredefined distance.
 8. The method of claim 1, wherein displaying theGUI that represents the current state of the vehicle and includes theselectable option comprises: displaying the GUI such that the selectableoption is selectable only for a threshold quantity of selections withina predefined duration of time.
 9. The method of claim 1, furthercomprising: receiving, from the vehicle, sensor data representing thesensor perspective of the environment; identifying one or more objectscausing the sensor perspective of the environment that is at leastpartially occluded based on the sensor data; and displaying the GUI withthe sensor perspective of the environment with boxes outlining the oneor more objects.
 10. The method of claim 9, further comprising:displaying the GUI such that the GUI includes a visual indication of thepredefined distance positioned virtually relative to the sensorperspective of the environment.
 11. A system comprising: a vehicle; anda computing device configured to: receive a request for assistance fromthe vehicle operating in an environment, wherein the request indicatesthat the vehicle is stopped at a location with a sensor perspective ofthe environment that is at least partially occluded; responsive toreceiving the request for assistance, display a graphical user interface(GUI) that represents a current state of the vehicle and includes aselectable option configured to enable the vehicle to gradually moveforward a predefined distance; and based on detecting a selection of theselectable option, transmit, to the vehicle, instructions that enablethe vehicle to gradually move forward the predefined distance, whereinthe vehicle is configured to gradually move forward the predefineddistance while also monitoring for one or more changes in theenvironment responsive to receiving the instructions.
 12. The system ofclaim 11, wherein the computing device is positioned remotely from thevehicle, and wherein the computing device and the vehicle communicatevia secure wireless communication.
 13. The system of claim 12, whereinthe computing device corresponds to a wearable computing deviceconfigured to simulate a point of view of the environment based on thesensor perspective.
 14. The system of claim 11, wherein the computingdevice is further configured to: receive sensor data representing thesensor perspective of the environment; and display the GUI thatrepresents the current state of the vehicle and includes the selectableoption such that the GUI includes the sensor perspective of theenvironment based on the sensor data.
 15. The system of claim 14,wherein the computing device is further configured to: obtain map databased on the location of the vehicle; and display an elevated view ofthe vehicle estimated based on the map data and the sensor data.
 16. Thesystem of claim 15, wherein the computing device is further configuredto: determine the predefined distance based on one or both of the sensordata and the map data.
 17. The system of claim 11, wherein the computingdevice is further configured to: receive a suggested distance forgradually moving forward from the vehicle; and based on the suggesteddistance, determine the predefined distance.
 18. The system of claim 11,wherein the computing device is further configured to: receive sensordata representing the sensor perspective of the environment; identifyone or more objects causing the sensor perspective of the environmentthat is at least partially occluded; and display the GUI with the sensorperspective of the environment with boxes outlining the one or moreobjects.
 19. The system of claim 18, wherein the computing device isfurther configured to: display the GUI such that the GUI includes avisual indication of the predefined distance positioned virtuallyrelative to the sensor perspective of the environment.
 20. Anon-transitory computer readable medium configured to storeinstructions, that when executed by a computing system, causes thecomputing system to perform operations comprising: receiving a requestfor assistance from a vehicle operating in an environment, wherein therequest indicates that the vehicle is stopped at a location with asensor perspective of the environment that is at least partiallyoccluded; responsive to receiving the request for assistance, displayinga graphical user interface (GUI) that represents a current state of thevehicle and includes a selectable option configured to enable thevehicle to gradually move forward a predefined distance; and based ondetecting a selection of the selectable option, transmitting, to thevehicle, instructions that enable the vehicle to gradually move forwardthe predefined distance, wherein the vehicle is configured to graduallymove forward the predefined distance while also monitoring for one ormore changes in the environment responsive to receiving theinstructions.